Digital video consumes large amounts of storage and transmission capacity. A typical raw digital video sequence includes 15 or 30 frames per second. Each frame can include tens or hundreds of thousands of pixels (also called pels). Each pixel represents a tiny element of the picture. In raw form, a computer commonly represents a pixel as a set of three samples totaling 24 bits. For instance, a pixel may comprise an 8-bit luminance sample (also called a luma sample) that defines the grayscale component of the pixel and two 8-bit chrominance sample values (also called chroma samples) that define the color component of the pixel. Thus, the number of bits per second, or bit rate, of a typical raw digital video sequence may be 5 million bits per second or more.
Many computers and computer networks lack the resources to process raw digital video. For this reason, engineers use compression (also called coding or encoding) to reduce the bit rate of digital video. Compression decreases the cost of storing and transmitting video by converting the video into a lower bit rate form. Decompression (also called decoding) reconstructs a version of the original video from the compressed form. A “codec” is an encoder/decoder system. Compression can be lossless, in which quality of the video does not suffer, but decreases in the bit rate are limited by the inherent amount of variability (sometimes called entropy) of the video data. Or, compression can be lossy, in which quality of the video suffers, but achievable decreases in the bit rate are more dramatic. Lossy compression is often used in conjunction with lossless compression—in a system design in which the lossy compression establishes an approximation of information and lossless compression techniques are applied to represent the approximation.
In general, video compression techniques include “intra-picture” compression and “inter-picture” compression, where a picture is, for example, a progressively scanned video frame, an interlaced video frame (having alternating lines for video fields), or an interlaced video field. For progressive frames, intra-picture compression techniques compress individual frames (typically called I-frames or key frames), and inter-picture compression techniques compress frames (typically called predicted frames, P-frames, or B-frames) with reference to preceding and/or following frames (typically called reference or anchor frames).
A frame (or other video picture) is typically represented as one or more arrays of pixel samples. For example, a YCbCr, or YUV, video data frame is represented as three planes of samples: a luminance (Y) plane of luminance samples and two chrominance (U, V) planes of chrominance samples.
Often in video compression techniques, blocks of pixel or spatial domain video data are transformed into transform domain data, which is often frequency domain (i.e., spectral) data. The resulting blocks of spectral data coefficients may be quantized and then entropy encoded.
When the data is decompressed prior to the resulting video being displayed, a decoder typically performs the inverse of the compression operations. For example, a decoder may perform entropy decoding, inverse quantization, and an inverse transform while decompressing the data.
Numerous companies have produced video codecs. For example, Microsoft Corporation has produced a video encoder and decoder released for Windows Media Video 8. Aside from these products, numerous international standards specify aspects of video decoders and formats for compressed video information. These standards include the H.261, MPEG-1, H.262, H.263, and MPEG-4 standards. Directly or by implication, these standards also specify certain encoder details, but other encoder details are not specified. These products and standards use (or support the use of) different combinations of the compression and decompression techniques described above. In particular, these products and standards provide various techniques to trade-off quality and bitrate for video data, including adjusting quantization, adjusting resolution (i.e., dimensions) of pictures, and frame dropping (i.e., temporal scalability).
While the compression techniques implemented in these products (or in compliance with these standards) are effective in many scenarios, it may be desirable to compress video data further than is allowable by a particular compression technique. For example, an upper limit on a quantization factor may limit the amount of compression that can be achieved by quantization. Moreover, in practice, extreme forms of quality degradation associated with particular techniques often limit how far video data may be compressed with those techniques. For example, large quantization step sizes often result in visible distortions such as blocking or ringing in displayed images. Excessive frame dropping typically leads to choppy video on playback.